Method and apparatus for variable accuracy inter-picture timing specification for digital video encoding

ABSTRACT

A method and apparatus for variable accuracy inter-picture timing specification for digital video encoding is disclosed. Specifically, the present invention discloses a system that allows the relative timing of nearby video pictures to be encoded in a very efficient manner. In one embodiment, the display time difference between a current video picture and a nearby video picture is determined. The display time difference is then encoded into a digital representation of the video picture. In a preferred embodiment, the nearby video picture is the most recently transmitted stored picture. For coding efficiency, the display time difference may be encoded using a variable length coding system or arithmetic coding. In an alternate embodiment, the display time difference is encoded as a power of two to reduce the number of bits transmitted.

CLAIM OF BENEFIT TO PRIOR APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 12/689,157, filed Jan. 18, 2010, now published asU.S. Publication 2010/0183073. U.S. patent application Ser. No.12/689,157 is a continuation application of U.S. patent application Ser.No. 11/621,974, filed Jan. 10, 2007, now issued as U.S. Pat. No.7,668,240. U.S. patent application Ser. No. 11/621,974 is a continuationapplication of U.S. patent application Ser. No. 10/792,669, filed Mar.2, 2004, now issued as U.S. Pat. No. 7,292,636. U.S. patent applicationSer. No. 10/792,669 is a continuation application of U.S. patentapplication Ser. No. 10/291,320, filed Nov. 8, 2002, now issued as U.S.Pat. No. 7,088,776. U.S. patent application Ser. No. 10/291,320 claimsbenefit to U.S. Provisional Patent Application 60/396,363, filed Jul.15, 2002. U.S. Publication 2010/0183073 and U.S. Pat. Nos. 7,668,240 and7,292,636 are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to the field of multimedia compressionsystems. In particular the present invention discloses methods andsystems for specifying variable accuracy inter-picture timing.

BACKGROUND OF THE INVENTION

Digital based electronic media formats are finally on the cusp oflargely replacing analog electronic media formats. Digital compact discs(CDs) replaced analog vinyl records long ago. Analog magnetic cassettetapes are becoming increasingly rare. Second and third generationdigital audio systems such as Mini-discs and MP3 (MPEG Audio-layer 3)are now taking market share from the first generation digital audioformat of compact discs.

The video media has been slower to move to digital storage andtransmission formats than audio. This has been largely due to themassive amounts of digital information required to accurately representvideo in digital form. The massive amounts of digital information neededto accurately represent video require very high-capacity digital storagesystems and high-bandwidth transmission systems.

However, video is now rapidly moving to digital storage and transmissionformats. Faster computer processors, high-density storage systems, andnew efficient compression and encoding algorithms have finally madedigital video practical at consumer price points. The DVD (DigitalVersatile Disc), a digital video system, has been one of the fastestselling consumer electronic products in years. DVDs have been rapidlysupplanting Video-Cassette Recorders (VCRs) as the pre-recorded videoplayback system of choice due to their high video quality, very highaudio quality, convenience, and extra features. The antiquated analogNTSC (National Television Standards Committee) video transmission systemis currently in the process of being replaced with the digital ATSC(Advanced Television Standards Committee) video transmission system.

Computer systems have been using various different digital videoencoding formats for a number of years. Among the best digital videocompression and encoding systems used by computer systems have been thedigital video systems backed by the Motion Pictures Expert Groupcommonly known by the acronym MPEG. The three most well known and highlyused digital video formats from MPEG are known simply as MPEG-1, MPEG-2,and MPEG-4. VideoCDs (VCDs) and early consumer-grade digital videoediting systems use the early MPEG-1 digital video encoding format.Digital Versatile Discs (DVDs) and the Dish Network brand DirectBroadcast Satellite (DBS) television broadcast system use the higherquality MPEG-2 digital video compression and encoding system. The MPEG-4encoding system is rapidly being adapted by the latest computer baseddigital video encoders and associated digital video players.

The MPEG-2 and MPEG-4 standards compress a series of video frames orvideo fields and then encode the compressed frames or fields into adigital bitstream. When encoding a video frame or field with the MPEG-2and MPEG-4 systems, the video frame or field is divided into arectangular grid of macroblocks. Each macroblock is independentlycompressed and encoded.

When compressing a video frame or field, the MPEG-4 standard maycompress the frame or field into one of three types of compressed framesor fields: Infra-frames (I-frames), Unidirectional Predicted frames(P-frames), or Bi-Directional Predicted frames (B-frames). Intra-framescompletely independently encode an independent video frame with noreference to other video frames. P-frames define a video frame withreference to a single previously displayed video frame. B-frames definea video frame with reference to both a video frame displayed before thecurrent frame and a video frame to be displayed after the current frame.Due to their efficient usage of redundant video information, P-framesand B-frames generally provide the best compression.

SUMMARY OF THE INVENTION

A method and apparatus for variable accuracy inter-picture timingspecification for digital video encoding is disclosed. Specifically, thepresent invention discloses a system that allows the relative timing ofnearby video pictures to be encoded in a very efficient manner. In oneembodiment, the display time difference between a current video pictureand a nearby video picture is determined. The display time difference isthen encoded into a digital representation of the video picture. In apreferred embodiment, the nearby video picture is the most recentlytransmitted stored picture.

For coding efficiency, the display time difference may be encoded usinga variable length coding system or arithmetic coding. In an alternateembodiment, the display time difference is encoded as a power of two toreduce the number of bits transmitted.

Other objects, features, and advantages of present invention will beapparent from the company drawings and from the following detaileddescription.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features, and advantages of the present invention will beapparent to one skilled in the art, in view of the following detaileddescription in which:

FIG. 1 illustrates a high-level block diagram of one possible a digitalvideo encoder system.

FIG. 2 illustrates a serious of video pictures in the order that thepictures should be displayed wherein the arrows connecting differentpictures indicate inter-picture dependency created using motioncompensation.

FIG. 3 illustrates the video pictures from FIG. 2 listed in a preferredtransmission order of pictures wherein the arrows connecting differentpictures indicate inter-picture dependency created using motioncompensation.

FIG. 4 graphically illustrates a series of video pictures wherein thedistances between video pictures that reference each other are chosen tobe powers of two.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

A method and system for specifying Variable Accuracy Inter-PictureTiming in a multimedia compression and encoding system is disclosed. Inthe following description, for purposes of explanation, specificnomenclature is set forth to provide a thorough understanding of thepresent invention. However, it will be apparent to one skilled in theart that these specific details are not required in order to practicethe present invention. For example, the present invention has beendescribed with reference to the MPEG-4 multimedia compression andencoding system. However, the same techniques can easily be applied toother types of compression and encoding systems.

Multimedia Compression and Encoding Overview

FIG. 1 illustrates a high-level block diagram of a typical digital videoencoder 100 as is well known in the art. The digital video encoder 100receives an incoming video stream of video frames 105 at the left of theblock diagram. Each video frame is processed by a Discrete CosineTransformation (DCT) unit 110. The frame may be processed independently(an intra-frame) or with reference to information from other framesreceived from the motion compensation unit (an inter-frame). Next, aQuantizer (Q) unit 120 quantizes the information from the DiscreteCosine Transformation unit 110. Finally, the quantized video frame isthen encoded with an entropy encoder (H) unit 180 to produce an encodedbitstream. The entropy encoder (H) unit 180 may use a variable lengthcoding (VLC) system.

Since an inter-frame encoded video frame is defined with reference toother nearby video frames, the digital video encoder 100 needs to createa copy of how decoded each frame will appear within a digital videodecoder such that inter-frames may be encoded. Thus, the lower portionof the digital video encoder 100 is actually a digital video decodersystem. Specifically, an inverse quantizer (Q⁻¹) unit 130 reverses thequantization of the video frame information and an inverse DiscreteCosine Transformation (DCT⁻¹) unit 140 reverses the Discrete CosineTransformation of the video frame information. After all the DCTcoefficients are reconstructed from iDCT, the motion compensation unitwill use the information, along with the motion vectors, to reconstructthe encoded frame which is then used as the reference frame for themotion estimation of the next frame.

The decoded video frame may then be used to encode inter-frames(P-frames or B-frames) that are defined relative to information in thedecoded video frame. Specifically, a motion compensation (MC) unit 150and a motion estimation (ME) unit 160 are used to determine motionvectors and generate differential values used to encode inter-frames.

A rate controller 190 receives information from many differentcomponents in a digital video encoder 100 and uses the information toallocate a bit budget for each video frame. The rate controller 190should allocate the bit budget in a manner that will generate thehighest quality digital video bit stream that that complies with aspecified set of restrictions. Specifically, the rate controller 190attempts to generate the highest quality compressed video stream withoutoverflowing buffers (exceeding the amount of available memory in adecoder by sending more information than can be stored) or underflowingbuffers (not sending video frames fast enough such that a decoder runsout of video frames to display).

Multimedia Compression and Encoding Overview

In some video signals the time between successive video pictures (framesor fields) may not be constant. (Note: This document will use the termvideo pictures to generically refer to video frames or video fields.)For example, some video pictures may be dropped because of transmissionbandwidth constraints. Furthermore, the video timing may also vary dueto camera irregularity or special effects such as slow motion or fastmotion. In some video streams, the original video source may simply havenon-uniform inter-picture times by design. For example, synthesizedvideo such as computer graphic animations may have non-uniform timingsince no arbitrary video timing is created by a uniform video capturesystem such as a video camera system. A flexible digital video encodingsystem should be able to handle non-uniform timing.

Many digital video encoding systems divide video pictures into arectangular grid of macroblocks. Each individual macroblock from thevideo picture is independently compressed and encoded. In someembodiments, sub-blocks of macroblocks known as ‘pixelblocks’ are used.Such pixel blocks may have their own motion vectors that may beinterpolated. This document will refer to macroblocks although theteachings of the present invention may be applied equally to bothmacroblocks and pixelblocks.

Some video coding standards, e.g., ISO MPEG standards or the ITU H.264standard, use different types of predicted macroblocks to encode videopictures. In one scenario, a macroblock may be one of three types:

1. I-macroblock—An Intra (I) macroblock uses no information from anyother video pictures in its coding (it is completely self-defined);

2. P-macroblock—A unidirectionally predicted (P) macroblock refers topicture information from one preceding video picture; or

3. B-macroblock—A bi-directional predicted (B) macroblock usesinformation from one preceding picture and one future video picture.

If all the macroblocks in a video picture are Intra-macroblocks, thenthe video picture is an Intra-frame. If a video picture only includesunidirectional predicted macro blocks or intra-macroblocks, then thevideo picture is known as a P-frame. If the video picture contains anybi-directional predicted macroblocks, then the video picture is known asa B-frame. For the simplicity, this document will consider the casewhere all macroblocks within a given picture are of the same type.

An example sequence of video pictures to be encoded might be representedasI ₁ B ₂ B ₃ B ₄ P ₅ B ₆ B ₇ B ₈ B ₉ P ₁₀ B ₁₁ P ₁₂ B ₁₃ I ₁₄ . . .where the letter (I, P, or B) represents if the video picture is anI-frame, P-frame, or B-frame and the number represents the camera orderof the video picture in the sequence of video pictures. The camera orderis the order in which a camera recorded the video pictures and thus isalso the order in which the video pictures should be displayed (thedisplay order).

The previous example series of video pictures is graphically illustratedin FIG. 2. Referring to FIG. 2, the arrows indicate that macroblocksfrom a stored picture (I-frame or P-frame in this case) are used in themotion compensated prediction of other pictures.

In the scenario of FIG. 2, no information from other pictures is used inthe encoding of the intra-frame video picture I₁. Video picture P₅ is aP-frame that uses video information from previous video picture I₁ inits coding such that an arrow is drawn from video picture I₁ to videopicture P₅. Video picture B₂, video picture B₃, video picture B₄ all useinformation from both video picture I₁ and video picture P₅ in theircoding such that arrows are drawn from video picture I₁ and videopicture P₅ to video picture B₂, video picture B₃, and video picture B₄.As stated above the inter-picture times are, in general, not the same.

Since B-pictures use information from future pictures (pictures thatwill be displayed later), the transmission order is usually differentthan the display order. Specifically, video pictures that are needed toconstruct other video pictures should be transmitted first. For theabove sequence, the transmission order might beI ₁ P ₅ B ₂ B ₃ B ₄ P ₁₀ B ₆ B ₇ B ₈ B ₉ P ₁₂ B ₁₁ I ₁₄ B ₁₃ . . .

FIG. 3 graphically illustrates the above transmission order of the videopictures from FIG. 2. Again, the arrows in the figure indicate thatmacroblocks from a stored video picture (I or P in this case) are usedin the motion compensated prediction of other video pictures.

Referring to FIG. 3, the system first transmits I-frame I₁ which doesnot depend on any other frame. Next, the system transmits P-frame videopicture P₅ that depends upon video picture I₁. Next, the systemtransmits B-frame video picture B₂ after video picture P₅ even thoughvideo picture B₂ will be displayed before video picture P₅. The reasonfor this is that when it comes time to decode B₂, the decoder will havealready received and stored the information in video pictures I₁ and P₅necessary to decode video picture B₂. Similarly, video pictures I₁ andP₅ are ready to be used to decode subsequent video picture B₃ and videopicture B₄. The receiver/decoder reorders the video picture sequence forproper display. In this operation I and P pictures are often referred toas stored pictures.

The coding of the P-frame pictures typically utilizes MotionCompensation, wherein a Motion Vector is computed for each macroblock inthe picture. Using the computed motion vector, a prediction macroblock(P-macroblock) can be formed by translation of pixels in theaforementioned previous picture. The difference between the actualmacroblock in the P-frame picture and the prediction macroblock is thencoded for transmission.

Each motion vector may also be transmitted via predictive coding. Forexample, a motion vector prediction may be formed using nearby motionvectors. In such a case, then the difference between the actual motionvector and the motion vector prediction is coded for transmission.

Each B-macroblock uses two motion vectors: a first motion vectorreferencing the aforementioned previous video picture and a secondmotion vector referencing the future video picture. From these twomotion vectors, two prediction macroblocks are computed. The twopredicted macroblocks are then combined together, using some function,to form a final predicted macroblock. As above, the difference betweenthe actual macroblock in the B-frame picture and the final predictedmacroblock is then encoded for transmission.

As with P-macroblocks, each motion vector (MV) of a B-macroblock may betransmitted via predictive coding. Specifically, a predicted motionvector is formed using nearby motion vectors. Then, the differencebetween the actual motion vector and the predicted is coded fortransmission.

However, with B-macroblocks the opportunity exists for interpolatingmotion vectors from motion vectors in the nearest stored picturemacroblock. Such interpolation is carried out both in the digital videoencoder and the digital video decoder.

This motion vector interpolation works particularly well on videopictures from a video sequence where a camera is slowly panning across astationary background. In fact, such motion vector interpolation may begood enough to be used alone. Specifically, this means that nodifferential information needs be calculated or transmitted for theseB-macroblock motion vectors encoded using interpolation.

To illustrate further, in the above scenario let us represent theinter-picture display time between pictures i and j as D_(i,j), i.e., ifthe display times of the pictures are T_(i) and T_(j), respectively,thenD _(i,j) =T _(i) −T _(j) from which it follows thatD _(i,k) =D _(i,j) +D _(j,k)D _(i,k) =−D _(k,i)Note that D_(i,j) may be negative in some cases.

Thus, if MV_(5,1) is a motion vector for a P₅ macroblock as referencedto I₁, then for the corresponding macroblocks in B₂, B₃ and B₄ themotion vectors as referenced to I₁ and P₅, respectively, would beinterpolated byMV _(2,1) =MV _(5,1) *D _(2,1) /D _(5,1)MV _(5,2) =MV _(5,1) *D _(5,2) /D _(5,1)MV _(3,1) =MV _(5,1) *D _(3,1) /D _(5,1)MV _(5,3) =MV _(5,1) *D _(5,3) /D _(5,1)MV _(4,1) =MV _(5,1) *D _(4,1) /D _(5,1)MV _(5,4) =MV _(5,1) *D _(5,4) /D _(5,1)Note that since ratios of display times are used for motion vectorprediction, absolute display times are not needed. Thus, relativedisplay times may be used for D_(i,j) display time values.

This scenario may be generalized, as for example in the H.264 standard.In the generalization, a P or B picture may use any previouslytransmitted picture for its motion vector prediction. Thus, in the abovecase picture B₃ may use picture I₁ and picture B₂ in its prediction.Moreover, motion vectors may be extrapolated, not just interpolated.Thus, in this case we would have:MV _(3,1) =MV _(2,1) *D _(3,1) /D _(2,1)Such motion vector extrapolation (or interpolation) may also be used inthe prediction process for predictive coding of motion vectors.

In any event, the problem in the case of non-uniform inter-picture timesis to transmit the relative display time values of D_(i,j) to thereceiver, and that is the subject of the present invention. In oneembodiment of the present invention, for each picture after the firstpicture we transmit the display time difference between the currentpicture and the most recently transmitted stored picture. For errorresilience, the transmission could be repeated several times within thepicture, e.g., in the so-called slice headers of the MPEG or H.264standards. If all slice headers are lost, then presumably other picturesthat rely on the lost picture for decoding information cannot be decodedeither.

Thus, in the above scenario we would transmit the following:D _(5,1) D _(2,5) D _(3,5) D _(4,5) D _(10,5) D _(6,10) D _(7,10) D_(8,10) D _(9,10) D _(12,10) D _(11,12) D _(14,12) D _(13,14) . . .

For the purpose of motion vector estimation, the accuracy requirementsfor D_(i,j) may vary from picture to picture. For example, if there isonly a single B-frame picture B₆ halfway between two P-frame pictures P₅and P₇, then it suffices to send only:D _(7,5)=2 and D _(6,7)=−1

Where the D_(i,j) display time values are relative time values. If,instead, video picture B₆ is only one quarter the distance between videopicture P₅ and video picture P₇ then the appropriate D_(i,j) displaytime values to send would be:D _(7,5)=4 and D _(6,7)=−1

Note that in both of the two preceding examples, the display timebetween the video picture B₆ and video picture video picture P₇ is beingused as the display time “unit” and the display time difference betweenvideo picture P₅ and picture video picture P₇ is four display time“units”.

In general, motion vector estimation is less complex if divisors arepowers of two. This is easily achieved in our embodiment if (theinter-picture time) between two stored pictures is chosen to be a powerof two as graphically illustrated in FIG. 4. Alternatively, theestimation procedure could be defined to truncate or round all divisorsto a power of two.

In the case where an inter-picture time is to be a power of two, thenumber of data bits can be reduced if only the integer power (of two) istransmitted instead of the full value of the inter-picture time. FIG. 4graphically illustrates a case wherein the distances between picturesare chosen to be powers of two. In such a case, the D_(3,1) display timevalue of 2 between video picture P₁ and picture video picture P₃ istransmitted as 1 (since 2¹=2) and the D_(7,3) display time value of 4between video picture P₇ and picture video picture P₃ can be transmittedas 2 (since 2²=4).

In some cases, motion vector interpolation may not be used. However, itis still necessary to transmit the display order of the video picturesto the receiver/player system such that the receiver/player system willdisplay the video pictures in the proper order. In this case, simplesigned integer values for D_(i,j) suffice irrespective of the actualdisplay times. In some applications only the sign may be needed.

The inter-picture times D_(i,j) may simply be transmitted as simplesigned integer values. However, many methods may be used for encodingthe D_(i,j) values to achieve additional compression. For example, asign bit followed by a variable length coded magnitude is relativelyeasy to implement and provides coding efficiency.

One such variable length coding system that may be used is known as UVLC(Universal Variable Length Code). The UVLC variable length coding systemis given by the code words:

1=1

2=0 1 0

3=0 1 1

4=0 0 1 0 0

5=0 0 1 0 1

6=0 0 1 1 0

7=0 0 1 1 1

8=0 0 0 1 0 0 0 . . .

Another method of encoding the inter-picture times may be to usearithmetic coding. Typically, arithmetic coding utilizes conditionalprobabilities to effect a very high compression of the data bits.

Thus, the present invention introduces a simple but powerful method ofencoding and transmitting inter-picture display times. The encoding ofinter-picture display times can be made very efficient by using variablelength coding or arithmetic coding. Furthermore, a desired accuracy canbe chosen to meet the needs of the video decoder, but no more.

The foregoing has described a system for specifying variable accuracyinter-picture timing in a multimedia compression and encoding system. Itis contemplated that changes and modifications may be made by one ofordinary skill in the art, to the materials and arrangements of elementsof the present invention without departing from the scope of theinvention.

We claim:
 1. A method comprising: receiving a plurality of encoded video pictures, wherein a particular encoded video picture is associated with an order value that represents a position of the particular video picture with reference to a nearby video picture; receiving a plurality of slice headers associated with the particular video picture, each slice header in the plurality of slice headers comprising an encoded instance of the same order value; and decoding the particular video picture by using the order value.
 2. The method of claim 1, wherein the order value represents a display time difference between the particular video picture and the nearby video picture.
 3. The method of claim 1 wherein the nearby video picture is an I video picture that comprises no macroblock that references another video picture.
 4. The method of claim 1, wherein said order value specifies a display order for the particular video picture in the plurality of video pictures.
 5. The method of claim 1, wherein each encoded instance of the order value is compressed by using variable length coding.
 6. The method of claim 1, wherein the particular video picture is a P video picture that comprises at least one unidirectional predicted macroblock and no bidirectional predicted macroblock.
 7. The method of claim 1, wherein the particular video picture is a B video picture that comprises at least one bidirectional predicted macroblock.
 8. The method of claim 7, wherein a motion vector for the bidirectional predicted macroblock is not interpolated.
 9. A method for decoding a plurality of video pictures of a video sequence, the method comprising: receiving a bitstream comprising an encoded first video picture, an encoded second video picture, and an encoded order value for the second video picture, the order value representative of a position of the second video picture with reference to the first video picture in the video sequence, the encoded second video picture comprising a plurality of slices, each slice associated with a slice header, said same order value encoded in each of the encoded second video picture slice headers; and decoding the second video picture by using said order value.
 10. The method of claim 9, wherein the order value represents a display time difference between the second video picture and the first video picture.
 11. The method of claim 9 wherein the first video picture is an I video picture that comprises no macroblock that references another video picture.
 12. The method of claim 9, wherein said order value specifies a display order for the second video picture in the plurality of video pictures.
 13. The method of claim 9, wherein each encoded instance of the order value is compressed by using variable length coding.
 14. The method of claim 9, wherein the second video picture is a P video picture that comprises at least one unidirectional predicted macroblock and no bidirectional predicted macroblock.
 15. The method of claim 9, wherein the second video picture is a B video picture that comprises at least one bidirectional predicted macroblock.
 16. The method of claim 15, wherein a motion vector for the bidirectional predicted macroblock is not interpolated.
 17. A decoder for decoding a plurality of video pictures of a video sequence, the decoder to execute a process comprising: storing a bitstream comprising an encoded first video picture, an encoded second video picture, and an encoded order value for the second video picture, the order value representative of a position of the second video picture with reference to the first video picture in the video sequence, the encoded second video picture comprising a plurality of slices, each slice associated with a slice header, said same order value encoded in each of the encoded second video picture slice headers; and decoding the second video picture by using said order value.
 18. The decoder of claim 17 wherein the first video picture is an I video picture that comprises no macroblock that references another video picture.
 19. The decoder of claim 17, wherein each encoded instance of the order value is compressed by using variable length coding.
 20. The decoder of claim 17, wherein the second video picture is a B video picture that comprises at least one bidirectional predicted macroblock. 